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ABSTRACT 

Colleges and universities are increasingly being held accountable for assessing and reporting 
student learning. Recently there has been increased focus on using assessment to improve 
learning over time. In this paper we present a simple, step-by-step assessment process that will 
deliver meaningful results to achieve these ends. We emphasize the need to conduct controlled and 
consistent pre- and post-testing so that the impact of pedagogical changes can be effectively 
measured. We explain how we have utilized this process in our Principles of Microeconomics 
courses to assess learning objectives and show that absolute performance on the assessment of a 
learning objective is not always consistent with relative performance improvement. These results 
reinforce the need to use pre-testing to effectively assess learning and measure the impact of 
pedagogical changes over time. 
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INTRODUCTION 



olleges and universities are increasingly being held accountable for assessing and reporting student 
learning. Accreditation agencies have been a big driver in this movement, but pressure is now also 
coming from the federal government, governing boards, and state legislatures (Myers, Nelson and 
Stratton 2009). There has additionally been a cultural shift in higher education towards assessment as a key element 
in student learning (Allen 2004). These changes have put pressure on departments and programs to develop 
assessment of learning processes to continually measure the extent to which they are meeting their learning 
objectives. 


Almost two-thirds of economics departments have a formal assessment plan, according to a national survey 
of economics department chairs in the United States (Myers, Nelson and Stratton 2011). Yet, there is little in the 
economics literature that departments can use to guide these efforts (Myers, Nelson and Stratton 2011). Two recent 
papers offer systematic approaches to assessment in economics at the programmatic level (Myers, Nelson and 
Stratton 2009) and course level (Cariglia-Harris and Hill 2010). While these papers offer valuable assistance to 
departments seeking to develop and implement assessment of learning strategies, neither paper addresses the 
importance of pre-testing. 

Pre-testing is essential for establishing a baseline. Without a measure of what students know at the 
beginning of a course or program, it is difficult to estimate the value added. Value-added approaches to assessment 
have become an important topic among education policymakers, researchers, and practitioners (Doran and 
Fleishchman 2005). In its 2006 report, A Test of Leadership: Charting the Future of U.S. Higher Education (U.S. 
Department of Education 2006), the Commission on the Future of Higher Education calls for colleges and 
universities to measure student achievement “on a “value-added” basis that takes into account students’ academic 
baseline when assessing their results” (p. 4). According to the American Association of State Colleges and 
Universities in its 2006 report, Value-Added Assessment: Accountability’s New Frontier , value-added assessment 
“speaks directly to the most important product of undergraduate education: the development of student knowledge 
and skills” (p. 3). 
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The purpose of this paper is to present a six step guide for developing an effective, value-added assessment 
of learning system. We illustrate this process by describing how we applied it in our Principles of Microeconomics 
courses, though this same process can be utilized to assess another course or program. The contribution of this paper 
is that it proposes a simple assessment process that utilizes pre-testing to establish a baseline to which post-test 
results can be compared. We discuss the importance of pre-testing for generating meaningful and comparable data 
over time as well as for reducing the incentives faculty may have to alter the assessment system to produce desired 
results. We discuss how consistency in assessment is vital for generating data that can best help departments 
implement and track the success of quality improvement initiatives over time to enhance learning. 

STEP 1: DEFINE LEARNING OBJECTIVES 

Identifying the key content you want to deliver in a course is the first step in developing an assessment of 
learning process. Ideally, the faculty members responsible for teaching the course will participate in this step. 
Starting with a discussion of how the course is taught and what material is covered, these individuals should 
construct of list of critical topics which are common across sections and most essential to the course. In our 
Principles of Microeconomics courses, for example, we engaged in such a process from which the list of critical 
topics presented in Table 1 emerged. 


Table 1: Critical Topics 

Marginal Analysis 
Opportunity Cost and the PPF 
Comparative Advantage 
Supply and Demand 
Elasticity 

Government Intervention 
Market Failures 
Production/Cost Structure 
Perfect Competition 
Monopoly 


Critical topics should then be grouped as appropriate into approximately 4 to 6 major areas of focus and 
formed into learning objectives. Learning objectives are broader than critical topics and communicate the 
overarching goals of a course utilizing action verbs. 1 We formulated the learning objectives presented in Table 2 for 
our Principles of Microeconomics courses. 


_Table 2: Learning Objectives_ 

LOl: Demonstrate how scarcity forces trade-offs, how every choice has an opportunity cost, and how different opportunity costs 
give rise to comparative advantages. 

L02: Apply knowledge of supply and demand, market equilibrium and elasticity to demonstrate how price allocates scarce 
resources. 

L03: Explain how markets reach efficient outcomes, why markets sometimes fail, and how government intervention can impact 
markets. 

L04: Describe a firm’s production process and resulting cost structure and show how output decisions are made. 

L05: Distinguish between perfectly competitive and monopolistic markets and understand the implications of each on market 
efficiency. 


Learning objectives can then be broken down into dimensions and even sub-dimensions if appropriate. 
Dimensions and sub-dimensions are narrower and more specific than learning objectives. As will be discussed in 
Step 2 below, the purpose of dimensions and sub-dimensions is to drill down a learning objective into more 
measurable units. Consider our first learning objective (LOl) in Principles of Microeconomics as an example. 
Within this learning objective, we defined the dimensions and sub-dimensions presented in Table 3. 


1 See Allen (2004) for a discussion of guidelines for writing effective learning objectives. 
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_Table 3: Dimensions and Sub-Dimensions_ 

Learning Objective: LOl: Demonstrate how scarcity forces trade-offs, how every choice has an opportunity cost, and how 
different opportunity costs give rise to comparative advantages. 

Dimension: LOl.l: Use a production possibility frontier to explain scarcity and opportunity cost. 

Sub-Dimension: LOl.1.1: Use a PPF to identify efficient, inefficient and unattainable production points. 

Sub-Dimension: LOl.1.2: Use a PPF to measure opportunity cost. 

Dimension: LOl.2: Comprehend comparative advantage and the gains from specialization and trade. 

Sub-Dimension: LOl.2.1: Compare the opportunity cost of two nations and determine comparative advantage. 

Sub-Dimension: LOl.2.2: Identify a mutually beneficial terms of trade price. 


Importantly, faculty responsible for delivering a course should review these learning objectives and 
approve them before moving forward in the development process. Getting faculty buy-in to any AOL process is 
difficult enough without having to contend with disagreement concerning the fundamental goals of a program or 
course. Faculty should agree with the list of learning objectives you have created. This is of critical importance 
because once the process is established it should not be changed. Changing the process will undermine the 
comparison of results over time. 

STEP 2: CREATE ASSESSMENT MEASURES 

Once learning objectives are well defined, you are ready to create measures to evaluate whether and the 
extent to which these objectives are being met. There are many ways to assess learning. 2 The type of measure that is 
best for your course depends on class size, structure and content. In our Principles of Microeconomics courses, for 
example, class size is so large that exams and problem sets consist of only multiple choice questions. Thus, 
assessment measures for this class are in the form of multiple choice questions as well. In other classes, it may be 
appropriate to utilize an essay, research paper, presentation or project to assess learning objectives. 

Assessment measures should target the sub-dimensions (or dimensions if no sub-dimensions exist) of 
learning objectives. This is the value of writing the sub-dimensions in the first place. Learning objectives are so 
broad that evaluating them with a single measure or with only a few measures can be difficult. Because dimensions 
and sub-dimensions are narrower and more specific, they are more easily assessed than the broader learning 
objectives to which they correspond. They lend themselves to the creation of focused measures. 

Sometimes it is useful to create multiple measures per sub-dimension. These is especially true when using 
multiple choice questions since these questions generally gather less information than an essay, for example, and are 
easily graded. However, it is important not to construct too many measures per sub-dimension because the AOL 
system you develop has to manageable. In Tables 4 and 5 illustrate examples of measures in the form of multiple 
choice questions and an alternative essay question for our Principles of Microeconomics classes. 

STEP 3: CREATE A PRE- AND POST-TEST STRATEGY 

An effective assessment of learning process should utilize some form of pre-testing in addition to post¬ 
testing. The value of pre-testing cannot be stressed enough. Pre-testing creates a baseline to which post-test results 
can be compared. By comparing students’ knowledge at two points in time we can better assess a course’s value 
added. Moreover, pre-testing allows us to control for student quality, both across institutions and over time 
(American Association of State Colleges and Universities 2006). As will be discussed in Step 6, this is particularly 
important when using an assessment system to measure the impact of curricular/pedagogical changes made in an 
effort to “close the loop”. Assessment programs that ignore pre-testing will have difficulty validating a continuous 
cycle of improvement process for which an assessment of learning process is designed. 


See Walstad (2001) for a discussion of assessment practices in economic education. 
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_Table 4: Multiple Choice Measures_ 

Learning Objective 

LOl: Demonstrate how scarcity forces trade-offs, how every choice has an opportunity cost, and how different opportunity 
costs give rise to comparative advantages. 

Dimension 

LOl.l: Use a production possibility frontier to explain scarcity and opportunity cost. 

Sub-Dimension 

LOl. 1.1: Use a PPF to identify efficient, inefficient and unattainable production points. 

Measure #1 

If a society is at a point inside its production possibilities frontier it implies that: 

a. technology is improving. 

b. all resources are used in production. 

c. more output could be produced using the same resources. 

d. goods are being produced with the optimal mixture of resources. 

Measure #2 

The production possibilities frontier shows: 

a. the products people want to have produced in a particular time period. 

b. the types of products that will be able to be produced in the future 

c. what an equitable distribution of products among citizens would be 

d. combinations of products that can be produced with currently available resources. 


_Table 5: Essay Question Measures_ 

Learning Objective 

LOl: Demonstrate how scarcity forces trade-offs, how every choice has an opportunity cost, and how different opportunity costs 
give rise to comparative advantages. 

Dimension 

LOl.l: Use a production possibility frontier to explain scarcity and opportunity cost. 

Sub-Dimension 

LO 1.1.1: Use a PPF to identify efficient, inefficient and unattainable production points. 

Measure 

Define and draw a PPF and identify production points that are attainable, unattainable, efficient and inefficient. 


In our Principles of Microeconomics courses we pre-test our students on the second day of class. Our pre¬ 
test consists of 20 multiple-choice questions, one for each sub-dimension of our learning objectives. We then embed 
these same 20 questions in the final exam for post-testing. It is important to try to establish post-test conditions that 
are similar to those of the pre-test. One potential criticism of our approach is that the incentives on an ungraded pre¬ 
test are not the same as on a final exam, resulting in an overestimation of the gains. While this may be true, it is not 
of significant concern when measuring the impact of pedagogical changes implemented to enhancing learning. 
When addressing learning enhancement we want measures that provide clean information on year-over-year relative 
gains. Using consistent, repetitive pre- and post-testing is a simple way to generate data that is meaningfully 
comparable over time, providing an ongoing baseline of information to effectively assess learning enhancement. 

STEP 4: DEFINE PERFORMANCE RUBRICS 

The fourth step in developing an effective AOL process is to construct performance rubrics. Rubrics are a 
set of criteria used to evaluate student performance on the measures written in Step 2. A rubric provides clear 
guidelines on how to categorize student performance into one of three categories: exceeds, meets or fails to meet 
expectations. A well designed rubric should lead to consistency across evaluators; rubrics should be specific enough 
that different evaluators would classify student performance similarly. The style of a rubric will vary depending on 
the type of measure. If, for instance, assessment measures are in the form of multiple choice questions, a rubric may 
specify how many multiple choice questions students must answer correctly in order to exceed, meet or fail to meet 
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expectations. If the assessment measure is instead in the form of an essay question the rubric will specify what the 
evaluator should look for in an answer in order to classify students into one of the three groups. Tables 6 and 7 
provide examples of performance rubrics for multiple choice measures and essay question measures that could be 
used in our Principles of Microeconomics classes. 


Table 6: Performance Rubric for Multiple Choice Questions 



Exceeds Expectations 

Meets Expectations 

Fails to Meet Expectations 

LOl 

Correctly answers 4/4 

Correctly answers 3/4 

Correctly answers less than 3/4 

L02 

Correctly answers 4/4 

Correctly answers 3/4 

Correctly answers less than 3/4 

L03 

Correctly answers 4/4 

Correctly answers 3/4 

Correctly answers less than 3/4 

L04 

Correctly answers 4/4 

Correctly answers 3/4 

Correctly answers less than 3/4 

L05 

Correctly answers 4/4 

Correctly answers 3/4 

Correctly answers less than 3/4 


_ Table 7: Performance Rubric for Essay Question _ 

Exceeds Expectations Meets Expectations Fails to Meet Expectations 

LOl.1.1 Can draw , define and label a PPF Can draw, define and label a PPF Cannot draw , define and label a PPF 

and use it to illustrate all of the and use it to illustrate three of the or use it to illustrate three of the 

following: attainable, unattainable, following: attainable, unattainable, following: attainable, unattainable, 
efficient and inefficient production efficient and inefficient production efficient and inefficient production 
points points points 


STEP 5: EVALUATE THE DATA 

How do you determine whether a learning objective has been met? Are you satisfied if at least 2/3 of 
students fall into the meets or exceeds expectations categories, or would you require at least 3/4? Targets may differ 
across learning objectives and even dimensions/sub-dimensions. Determining acceptable benchmarks can be 
difficult and is highly subjective. Moreover, evaluating data based on benchmarks alone creates an incentive for 
faculty to tinker with the level of difficulty of their assessment measures in order to hit certain targets. 

In fact, when an assessment process is used for continual improvement a benchmark itself is less important 
than whether your numbers increase over time. And, if courses or programs are evaluated based on improvements 
over time rather than absolutes, faculty have less incentive to change the assessment measures in order to hit certain 
targets. Here again the need for pre-testing is apparent. Pre-testing controls for changes in student quality over time 
so that we can better assess the institutional impact of the course or program. Moreover, pre-testing controls for the 
difficulty of the assessment measures and therefore reduces the incentive faculty have to construct easier measures 
since the focus is on value-added. 

Consider the results from our Principles of Microeconomics courses presented in Table 8. The first row 
reports the post-test percentages of students who answered at least three out of four multiple-choice questions 
correctly for each of the five learning objectives and lists rankings based on these percentages. Figure 1 depicts 
these results and demonstrates that the largest percentage of students met expectations for LOl and the smallest 
percentage met expectations for L04. However, when we look at the second row of Table 8 we see that baseline 
pre-test performance is dramatically different across LO’s. This suggests that some LO’s had multiple-choice 
questions that were either easier to guess or were based on concepts that students had some intuitive familiarity with 
prior to taking the course. The third row of Table 8 compares the pre-test percentages to the post-test percentages. 
Figure 2 illustrates this comparison and shows that the greatest value-added was made for L03 and the least for 
L02. 


A simple evaluation of Figures 1 and 2 together reveals that absolute performance is not a good measure of 
value-added. Although absolute performance is highest for LOl, value-added is greatest for L03. Additionally, 
while L02’s absolute performance is second highest, its value-added is lowest. These results demonstrate that 
having data on both absolute performance and pre-post gains gives a much clearer picture of learning. This is 
especially important when using the assessment process to “close the loop” as will be discussed next in Step 6. 
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Table 8: Results 


LOl 

L02 

L03 

L04 

L05 

POST-TEST 

% Meets or Exceeds Expectations 

65.83% 

54.17% 

46.25% 

45.83% 

49.58% 

Rank (based on absolute %) 

1 

2 

4 

5 

3 

PRE-TEST 

% Meets or Exceeds Expectations 

14.17% 

26.67% 

6.25% 

12.08% 

7.92% 

Rank 

2 

1 

5 

3 

4 

PRE-POST 

Percentage Points Gained 

51.66 

27.5 

40 

33.75 

41.66 

Percentage Gain 

365% 

103% 

640% 

279% 

526% 

Rank (based on % gain) 

3 

5 

1 

4 

2 


N=240 
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The value of assessment of learning is not just the determination of whether learning objectives are met but 
importantly the use of this information to develop a process for continual improvement. In this final step, you will 
develop and implement a plan for improvement if a learning objective is not met and then re-asses the learning 
objective to evaluate the impact of your plan. This critical step of your assessment of learning process closes the 
loop 


If your assessment data reveals that a learning objective is not being met, develop an action plan of content, 
pedagogical or curricular changes designed to improve performance. Sub-dimensions can be helpful in this step. 
Because they are narrower than learning objectives, they will better reveal where specifically a learning objective is 
not being met and help you formulate a plan for improvement tailored to the specific skill of the sub-dimension. 
Sometimes an action plan will focus on the course you are assessing. However, it is possible that the changes need 
to occur in another course. Once more, this is a benefit of pre-testing. Since we can measure what students know at 
the beginning of a course or program, we can get a sense for whether changes need to be made before a 
course/program in order to address any deficiencies. 

Lastly and importantly, evaluate the impact of your changes in the next assessment loop. Determine 
whether your action plan improved performance in meeting your learning objective. If it did, continue to incorporate 
the change. Are you still not meeting expectations? If so, develop another action plan to make improvements. 
Remember that assessment in an ongoing process. We as educators should be regularly evaluating the extent to 
which we are meeting our goals and making changes as necessary to improve. 

CONCLUSION 

In this paper we outlined a methodology for the development of a simple and effective assessment protocol 
that provides meaningful results that can be used to evaluate and enhance learning over time. Our six step process 
begins by asking the question, “What do we want our students to learn?” and then illustrates how to develop 
learning objectives, assessment measures, and rubrics with the ultimate goals of determining whether our students 
are learning what we want them to learn, finding out where we can improve, and evaluating pedagogical or 
curricular changes designed to improve performance. 

We also discuss the benefits of pre-testing to measure value-added as opposed to absolute performance 
measures. We believe the importance of implementing a consistent and repeated pre-post testing procedure cannot 
be emphasized enough. Meaningful relative comparisons are critical to accurately measuring the outcome of 
changes in pedagogy over time. Without such a protocol it will be nearly impossible to effectively demonstrate the 
successful “loop-closing” cycles of continuous improvement that assessment agencies increasingly require. 

We understand the resistance to assessment of learning in higher education. However, assessment of 
learning is here to stay. Instead of resisting this process, we believe it is important to embrace it and change the view 
of assessment as a bureaucratic requirement to a valuable pedagogical tool. 
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